1 link tagged with all of: evaluation + synthetic hospital + clinical nlp
Click any tag below to further narrow down your results
Links
Researchers created a fully synthetic but realistic electronic health record dataset with 1,268 patients and 5,602 encounters, designed to let AI models be tested on clinical tasks without privacy concerns. Physicians couldn't reliably distinguish it from real charts, but current language models still fall short—the best achieved an F1 score of 0.73 on reconstructing patient problem lists, below the average physician's 0.89.
- The dataset solves two major barriers: it's openly shareable (no protected health information) and has verifiable ground truth (every diagnosis and finding is grounded in standard medical ontologies and traced back to source materials)
- The benchmark is genuinely difficult—top models miss roughly half of clinically relevant findings when summarizing charts and don't approach human physician performance
- Physicians in blinded review distinguished synthetic records from real ones at near-chance rates (53%), validating the realism of the dataset